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ABSTRACT ^ ^ 

Project "Freestyle" involved the development of 
prototypical television and print materials intended to comfcat 
sex-role stereotyping in career-related attitudes of nine to 
twelve-.year-old children. In the first 16 months of the project an 
assessment' was made of tbe reactions to three pilot stovs^ among 
students, teachers, administrators, a,nd parents at four test sites. 
This paper describes the sample, research design, procedures, and 
instruments used to obtain information about student characteristics 
and reactions to the materials, teacher ratings of their own and 
perceived student attitudes toward the material:S, and administrators* 
and parents' reactions. The paper t hen outlines a number of analysis 
strategies used in gathering and processing \ the data. Among the 
topics discussed are the decision to use, keypunching, the preparation 
phase involving a series of "dummy" data computer runs, and 
statistical examination of the data. The analysis strategies were 
aimed at assessing students* liking and comprehension of the three 
diffirrent pilots, the effects of combining '(Jrint materials with 
television materials, students' comprehension of the program 
objectives as presented in different formats, and preferences of 
teachers, administrators, and par^ents .among the pilot programs. 
(Gi) 
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Design and Procedur?es ' ' 1 April 2b, 1978 

.. ^. ■ ^ ■ . 

In the first 16 months of the project the aim was ''the 
development of prototype television "materials a School 
Guide which' would' suggest classroom activities to complement' 
the Tnat#rials a Home Guide having a similar' purpose and a 
CQmic B(^*^ which would reinforce thfe ser^ies objectives 
through an alternative medium popular y;ith children 'in this 
age range, • / 

^ The emphasis was still upon foripative research; henge 
th^e was interest in obtaining a variety of responses to 
the mater ials ''m addition to corrprehens ion of the basic 
objectives. For instance owe wanted to aet;ermine students 
likabiiity of the materials and also their, cpmprehension of 
the factual' content which was used as a vehicle to 
communicate the spec if^"^ ob ject/Tves. Also, there was a 
concern with gathering data which would reflect upon the 
possible utiliza^tioh of the series and information which 
might b# used to. increase -potential for utilization. 
However, there was no attempt in the present study to assess 

attitudinal change as a consequence of exposure to the 

... ^-^ \ 

materials since such change had always been envisioned as 

cumulative effect of seeing the entire series rather than an 

indiviaual television segment or nonbroadcast activity. 
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The Sample " ' * 

The sample consisted of 1639 stuaents^ 60 teachers^ and 
11" administrators plus 83 parents at four test sites 
(Patterson,) New Jersey; Lexington^ Kentucky; .-LincopLn, 
Nebraska; Austi» ana Lockhart, Texas). Students 
comprised of five intact Ci^issrooms at each of the three 
sfchools w^ithin eac"h site/- Classrooms wer'^ chosen to 

represent variations in hex^ 'grade level^, and^ ethnicity as 

^ t ' ^ ' 

well as ' urban-rural and socio-economic contras.ts, 

dexican-Amer ic^n Spanish-speaking ' students (Te^as) and 

Puerto Rican-Amer ican Spanish-speaking stu^^nts (New* Jersey) 

were included in the sample. (Classrooms were not randomly 

selected^ ^ but chosfen as a sampl^of hypothetical population 

variations based on the^se demographic, character istics ). All 

students who savy^at 'least one fifteen-minute pilot s^^ent 

wer^ included in the sample. 

Research Design ' • ' * 

V 

Evaluation was conducted in' a quasi-experimental 
design^ with two treatment conQitionsr 1) full treatment 
which included the televisi^rT ^pilots plus nonbr oa'd'cas t 
materials and 2) par t ial Cixeatment which included only the 
televis^ion pljots. . • ' 

The full treatment group receiyed the ^chool Guide and 
the Comic Book for classrootn use^ a ^Parent Quide to send 
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pilQts (one questionnaire per pilot), 2) personality and 
' , activity ratings for' audience segmentation purposes^ 3) 
reactions to the conic book {fuM treatment subjects only) 
4) opinion^ media use^ audience profile , itenfs^ •and 
preference from amt)ng the three pilots. Each instrument for 
segment testing had items drawn from a previously developed 
^ pool on the basis of s tat is tic a 1 reliability frbm pce-test 
results. As a little aside here;, we were concerned with the 
i^sue of' identifying students across five or six 
questionnaires/ si ne'e* we would be making comparisons of data 
»acrc%s three pilots.. Rather than relying on students or 
te^cher^s to m.ake 'corr.ect assignments from, day to day we ' 
bo.th pre-numbered and colors-coded our sets of student 
questionnaires and inserted them^ in folders which the 
teachers distributed then dollected for each test period. 
The * preparations worked since we did not lose a single 
student* ^ - ' , ' 

* Three instruments were developed to gather information 
from^ teachers. A post-vieVihg survey^ contained questioi^ 
regarding utiirzation of the three pilots^ teacher ratings 
of, penceived student attitudes tToward the show^- as well as 
tea.cher ratings of their own attitudes toward the objectives 
of the ser ies . Full treatment teachers also completed 
qu^st ionnairefe regarding their perceptions of the teacher 
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pilQts (one questionnaire per pilot), 2) personality and 
• . activity ratings for' audience segmentation purposes^ 3) 
reactions to the conic book (fuM treatment subjects only) 
4) opinion^ media use, audience profile ^ itenfs, *and 
preference from amt)ng the three pilots. Each instrument for 
segment testing had items drawn from a preViouely developed 
^ pool on the basis of statistical reliability frbm pce-test 
results. As a little aside here;, we were concerned with the 
* i^s.ue of' identifying students across five or six 
questionnaires/ since* we would be making comparisons of data 
, across three pilots.. Rather than relying on students or 
te^cher^s to m.ake 'corr^ect assignments from, day to day we 
^ both pre-numbered and colors-coded our sets of student 
questionnaires and inserted them^ in folders which the 
teachers distributed then dollected for each test period* 
The * preparations worked since we did not lose a single 
student. • - . ' - • 

• Three instruments were developed to gather info r mat ion 
f r om^ teachers , A post-vie>^i'ng survey^ contained quest io^^ 
regarding utilrzation of the three pilots^ teacher ratings 
of , pence ived student attitudes tToward the show^- as well as 
tea.cher ratings of their bwn attitudes toward the objectives 
of the ser ies . Full treatment teachers also completed 
qufest ionnair efe regarding their perceptions of the teacher 
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POPULATION I INSTRUMENTS 


1 

i DATA COLLECTION POINTS ' ^ 


, LENGTH 


.^-^ * ' CONTENT 


j 

Students j Ftm 4 

L . Forn 1 
^ form 2 
j. * Forn 3 
1 ^ Form* 5 

^ 1^ ^ Form 5 


Prior to segment viewihn 
Fonowing each 15-minute segment 

Follov/lng comic book 
1 Following all pilot activities 


10 min. 
10 min. 

10 m1n. 
5 m1n. 


ireacher-admini stered cuGstionnairp^l 
Ksychographic Instrument < • 

i^pYkeability, coma>:^h«ns1on, opinion • 

luCTlS 

Comprehensive follow-yp Items; opinlor 
media use, a\jd1en,ce profile Items • 


Teachers - ! 

i 

t ■ 

j Form 8 
i Form • 7 > 

— J — ^ ■ ( 


• 

rollowlnq all pilot a^tfvlties 
Followtng all -pilot activities 

, m . n\ ' - k. 


10 min. 
10 min. 


* 

School Guide items 
CompreWnslve follow-up Items 


Administrators | Form 9 


I - " V - ■ ' — ^ 

Fo'llowinn aU pilot activities 


' 10 min. 


Y-^^ -__ 

Comprenenslve follovz-up Items . 

— _ 


Parents^ 

>• 


" Form 10 
' Parent Home Survey 


'-^ 

Followinnj^an pjlot activities ^. 

F(>nov/1nq home viewing' 


IT- 

' .20 min. 

' 5 min. 


Home Guide Jlems and Comcrehenslve 
follow-up Items ^ \ : 
Home ^feulde' Items and Comprehensive* ' 
follow-up Items 


All .of the above 

_ERIC • 




TKIrd v/eek In October 

,# 


1'5 niin. 


Field consultants will Interview at 
Veast 1 teacher and 1 admlnstrator 
from each school , student from each of 
the 3 graxie^and 1 family from each ' 
school uslng^an open-ended format. 
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<^ " 

.workshops and one-page ratings for each activity they 
conducted in their classrooms. 

Administrators completed a single form concerning series 
utilization in which they were requested - to rate the 
FREESTYLE package on its apppropr lateness^ acceptability to 
the community^ and the degree H:o which they would recommend 
package use to their teachers. 

Two samples of parents filled out one parent 
questionnaire each. Par t icipa.nts in ffbrent panels formed^ 
by schools in con3unction with the consultant., received the 
Parent Survey^ Form 10. This covered aspects of the 
FREESTYLE series and the Home Guide. A ''shorter form was 
sent home via students for parents to complete and return. 



Procedures 



/ 



Inservice' training was provided for full treatment 
teachers^ during the first week in October^ An informal 
orientation to the project and the i|se of the evaluation 
materials was given by the site Consultants to the partial 
treatment teachers. 



At most sites the f if teen-minute segments were v^^ewed on 
an every-other-day schedule with first and second segments 
of each pilot to be shown in order^ but with the order of 
the pilots randomized* The .comic books were assigned so 



0 
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that half ot the /school^s Osfed each alternatfve -version 
(Realistic or Cartoon art style). The teachers distiributed 
the comic book atter children viewed the first pilot with 
instruct ions -for the children to read it at home 

\' ■ ■ ■■■ ^ 

Analys is St rat eg ies 

. My purpose is not to present a cookbook of statistical 
analy.sis^ . but ' rather "to give you an idea of 'how .we coped 
with a tremendous amount of data and a. very short turnaround 
time. , ^ * 

To illustrate the magnitude of* the tasK^ oUr 
respondents resulted in four data sets with information from 
students teachers^ adpiaistr ators^ and parents^ For the. 
Students alone^ six questionnaires meant 9834 cards and 
about 155^000 data points. VJe began testing in October of 
last year^ and the last set of questionnaires was due around' 
the first week of November* • This gave us only "about three 
we'eka to sort, check; punchy and analyze the results and to 
^write our report before we presented the data" on November 
28. _ , 

>, 

In addition to our_ large data set aqd our 'time 
restraints^, we Jiad another, kind of impetus, to perform fast> 
turnaround of the^data, The production staff at KCET had a 
schedule to meet an^ we had an obligation to deliver 'result^" 

\ 
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to them as soon as possible befora they "made the crucial 

d^ecisions for -the final series^ They could not wai-t for six 

month© while we performed endless data checik.^^- and did 
ana!uyses ad inf initum > ^ ' • , * 

Clearly.^ -we needed a strategy ^ Vvhat .. would be"' most 
helpful to us when the locust-like clouds of ^ quest ionnaire^ 
from the. Midwest and elsewhere began to drop on us ' via air 
freight? 

The answer was clear, Pre-planning .v/oulS serve the 
task. We did every thip^ possible *bef ore the q\iest ^.onnair es- 
were ^ even sent to the sites After th^e first shipment (the 

A 

xfirst wave of oata) came back to us^ we evep filled in our 
tables and wrote our report beforehand. This is how it 
worked . ^ - • ' . . > - ■ 

Several' months before the pilots w/ere viewed We began 
our preparations* I will " deaJL primarily here with .the 
student sample as it presented the greatest, logistic and 
processing challenges, ^ /-C / • - * 

W-eS b'egan even before the (Questionnaire draft's were 
available. One decision that had to be made at this ooint 



1' 



was the method to be used in gathering and processing the 
data*. ' We considered an opt ical ^ scanning approach but 



^ecidea^that in terms of form restrictions^ error-rate -and 
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cost Having questionnaires keypunched would be the more 
efficient meth'o.d^ * ^ ^. ..r 

Once^ we decided to use keypunching^ we had to determine 
what a^ tolerab^yerror rate might be. This was necessary 
for two, reasons-. One, because- with fast d^'fca turnaround we 
cou'ld not .realistically , expect to check ev^ry -data point 
with its quest i^/ina ire counterpart- Two, "an agreement with 
fcjie ^keypunch company regarding quality prior to delivery of 
~ihe material' vfis^res a quality guarantee. 



•J 



^ ^ We deterrnined that we would teel secure with a ,5 ^er 
^^^^ error rate. We woula check punched^ car ds in, two ways 
to assurer oursejves t hi t 'this hacT not been exceeded*- .F.irst/ 
a list wOulo^Be raaa^ of all the punched cards- so -we could, 
chfeck ^v'ery tenth- one point by point. Second, we woutd ' 
develop'roTtran prc^gt^s to search for punches" lying outside 
ttie valid range \f pCincheS and to search for duplicate cards 
(these occur ocr^asi^n^y wnen%^rds are verifi'e^), 

^ ^ 0,n.ce the^ quest ionnaire' drafts were ready^ we couid 
becjin to wr^ite the -Fottran* progra)s>s just pientioned as well 
- /as t'he ,^rogram3^#l%hat we 'would use'' t'o^ analyze the data^ 

- inyolveci a - series 5^^ot /'*^dumiiiy*| aata computer runs for the 
^ . ' » ,\ » ' * ' ' ^ 

« * . . 1 • . ; - . ' - 

\ \ ■■■■ : ■ i . 
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child sample in which keypunch quality^ our process^ and our 
progra»r.s were checked. Data were-created which approximated 
the responses we anticipated and represented ' the sample 
demogr aphical]^ ; We knew precisely what we pu^ into the 
system so we knew also precisely wffat we should get out-. 

We also created dummy report drafts which included 
tables ready to be filled in wit^h data and as much of the 
^report as we could draft at that time (one example is the 
^Methods section), When the questionnaires were sent out, we 
were ready. Our computer programs and tables were waiting^ 

As soon as the data began to arrive from the national 
sites^ they were sent to be keypunched after being first 
counted and checked. We also coded the open-ended responses 
accoraing to a pre-determined system . 

The- dafca was analyzed as soon as it returned from the 
keypuncher* By the middle of October we had data from all 
questionnaires . although ^t was confounded by site- Results 
from this first wave both surprised and pleased us;. as the 
response patterns ^never cha^il-ged as the amount of data 
increased ' 

Statistical examination of the data involved the total 
sample as well as comparisons between the subsets of sex, 
^thnic group, graae^ and site. Grade , for i each child was 
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assigned by teacher- rather than .self-report in- order to 
increase precision. We also examined the effects of varying 
piiot presentation oruer f anci although sone confounding with 
sites existed^ ife appearea to make no difference in the 
overall results which pilot the children saw first,- s'econd, ♦ 
or third ♦ ' ^ 

In order to answer the research questions several 

techniques were used. Comparison between pilots on 

■J 

likability, co:r>prehens ion ^ and preference was done primarily 
with T-tests for non independent measures- Assessment of\ 
differences between versions of the comic book was performed 
in the same manner » We compared the success with which each 
pilot segment achieved its curriculum objective by plotting 
histograms of mean percentage .of comprehensioT^ and comparing 
these visually with the overall mean on objective 
comprehension. Administrator reactions were also reported 
as means with no significance tests due to the very low 
number ^f administrators responding. 

When we comipared responses of sample subgroups we used 
indepenoent group T-tests. When we were interested in 
look ing at the relationships between two or more independent 
measures, analysis of variance was performed ^ through 
multiple regress ion, which freed us of the necessity for 
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( 

In setting our alpha level^ ,05 was initially used. It 
soon become obvious^ however^ that with our rather large 
child saraplef differences were showing 'significance which 
woula not allow us to present the results meaningfully 
,1nerefore, we raised the a'lpha level to ^01^ 



The entire analysis strategy was aimed at^ answering 
four primary research questions. These were: 

1 ) Television foirTiat 



What are the relative contrasts in students likability 
and comprehension of; and preference for, the' three 
different pilots? 



2 ) iNonbr oadcas t mater ials 



U'hat are the effects of * combining nonbroadcast 
materials with tne television materials? What are the 
differences in the two art forms of the Comic Book? 



3) ' Comprehension of Curriculum Objectives 



What is the relative comprehension of the different 
objectives as presented in different formats? 



4 ) Ut i 1 izat ion 



\ 
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To what degree was the series responded to favorably by 
teachers, administrators and parents? What were t^ir 
relative preferences among the pilots?" 

Results to answer these questions were kept strictly 
se|cret until the day of the consortium report in order to 
prevent Production from jumping to conclusions on the basi^ 
of partial inf orma^tToiT, An entire day was sched^iled for the 
report of the results and ensuing discussion. What wle^found 
is discffesed in a subsequent paper, 

\ - 

\ 




